[8.19] (backport #9048) ci: build agent from snapshot DRA by mergify[bot] · Pull Request #9280 · elastic/elastic-agent

mergify · 2025-08-07T08:44:27Z

What does this PR do?

This PR reworks the .package-version file format and integrates it into our CI and mage workflows to enable deterministic, snapshot-based builds of Elastic Agent based on published DRA (Downloadable Release Artifacts).

Key changes:

.package-version is now a structured JSON file containing version metadata (version, build_id, stack_build_id, manifest_url, etc.).
CI scripts and mage targets now conditionally derive the agent version and build metadata from .package-version when USE_PACKAGE_VERSION=true is set.
The mage target UpdatePackageVersion replaces previous logic with a structured update mechanism for .package-version.
CI integration steps now consistently use the snapshot version and manifest provided in .package-version.

Why is it important?

We previously had a fundamental dependency issue in the release flow: to produce a new Elastic Agent DRA, we needed to bump the agent version — but our CI integration tests required that same DRA to already exist in order to pass. This created a timing problem and made the release workflow fragile.

With this PR:

The CI system and mage targets now honour .package-version when USE_PACKAGE_VERSION=true is set, allowing packaging and testing to rely on a known-good, previously published DRA.
This ensures that builds and tests only proceed against DRAs that have been published by the unified release process, making test results and releases more predictable and reliable.
PRs such as #9163, which update .package-version, now become the authoritative source of truth for the Elastic Agent version used in the repo and must be handled with utmost care.
Note: Even if a component publishes its own DRA, Elastic Agent will only package it once the unified release has run and the respective PR to update the .package-version has been merged.

This enables a reproducible and version-pinned CI system while allowing us to automate snapshot bumps in a controlled and testable way.

An experimental version bump under commit a57ee10 verified this flow successfully. The CI run is available here.

Checklist

I have read and understood the pull request guidelines of this project.
My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in ./changelog/fragments using the changelog tool
I have added an integration test or an E2E test

Disruptive User Impact

None expected. Existing workflows are unaffected unless USE_PACKAGE_VERSION=true is explicitly set. In that case, .package-version is required and must be up to date.

How to test this PR locally

specify USE_PACKAGE_VERION=true and call any mage target e.g.

USE_PACKAGE_VERSION="true" EXTERNAL="true" PLATFORMS="linux/arm64" PACKAGES="docker" DOCKER_VARIANTS="basic" mage package

Related issues

N/A

This is an automatic backport of pull request #9048 done by [Mergify](https://mergify.com).

mergify · 2025-08-07T08:44:30Z

Cherry-pick of a155660 has failed:

On branch mergify/bp/8.19/pr-9048
Your branch is up to date with 'origin/8.19'.

You are currently cherry-picking commit a155660c4.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   .buildkite/integration.pipeline.yml
	modified:   .buildkite/scripts/buildkite-integration-tests.ps1
	modified:   .buildkite/scripts/buildkite-integration-tests.sh
	modified:   .buildkite/scripts/buildkite-k8s-integration-tests.sh
	modified:   .buildkite/scripts/steps/ess.ps1
	modified:   .buildkite/scripts/steps/ess.sh
	modified:   .buildkite/scripts/steps/ess_start.sh
	new file:   .buildkite/scripts/steps/integration-cloud-image-push.sh
	modified:   .buildkite/scripts/steps/integration-package.sh
	modified:   .buildkite/scripts/steps/integration_tests.sh
	modified:   .buildkite/scripts/steps/integration_tests_tf.ps1
	modified:   .buildkite/scripts/steps/integration_tests_tf.sh
	modified:   .github/workflows/bump-agent-versions.sh
	modified:   dev-tools/mage/manifest/manifest.go
	new file:   dev-tools/mage/packageversion.go
	modified:   dev-tools/mage/settings.go
	modified:   magefile.go
	modified:   test_infra/ess/deployment.tf

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   .package-version
	both modified:   testing/integration/ess/upgrade_broken_package_test.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

elasticmachine · 2025-08-07T08:44:45Z

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

pkoutsovasilis · 2025-08-07T08:56:54Z

first I wanna see how this PR behaves on main and if everything checks out, I am gonna deal with the backports

* feat: rework .package-version and mage integration:UpdatePackageVersion to make CI build always from snapshot DRA * feat: incorporate USE_PACKAGE_VERSION in mage * experiment: bump version.go * Revert "experiment: bump version.go" This reverts commit a57ee10. * chore: bump .package-version * feat: allow AGENT_VERSION to be overridden by env var * fix: use named args for all args in integration_tests_tf.ps1 * feat: panic on err of initPackageVersion * fix: don't panic when .package-version file doesn't exist, log it instead * feat: rework fabrication of CI_ELASTIC_AGENT_DOCKER_IMAGE * feat: use os.WriteFile in writePackageVersion * chore: bump to latest snapshot DRA * fix: always DownloadManifest if PackagingFromManifest is set in mage package * fix: check err of filepath.Abs(dropPath) (cherry picked from commit a155660) # Conflicts: # .package-version # testing/integration/ess/upgrade_broken_package_test.go

pkoutsovasilis · 2025-08-07T21:44:58Z

Blocked by #9252

elasticmachine · 2025-08-07T21:54:32Z

💔 Build Failed

Buildkite Build
Commit: d0c48ee

Failed CI Steps

History

💔 Build #24894 failed e99e0d5
💔 Build #24882 failed 70c1156

cc @pkoutsovasilis

elastic-sonarqube · 2025-08-08T14:54:41Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

mergify bot requested a review from a team as a code owner August 7, 2025 08:44

mergify bot added backport conflicts There is a conflict in the backported pull request labels Aug 7, 2025

mergify bot requested a review from a team as a code owner August 7, 2025 08:44

mergify bot requested review from pchila and ycombinator and removed request for a team August 7, 2025 08:44

mergify bot assigned pkoutsovasilis Aug 7, 2025

mergify bot mentioned this pull request Aug 7, 2025

ci: build agent from snapshot DRA #9048

Merged

8 tasks

github-actions bot added Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team skip-changelog Testing:run:TestUpgradeIntegrationsServer labels Aug 7, 2025

pkoutsovasilis mentioned this pull request Aug 7, 2025

chore: remove agent version pinning in the CI #9225

Closed

8 tasks

pkoutsovasilis added 4 commits August 8, 2025 00:43

fix: resolve conflicts

1df5782

fix: remove redundant agent version pinning

3464003

fix: bump beat to the commit of the snapshot DRA in .package-version

d0c48ee

pkoutsovasilis force-pushed the mergify/bp/8.19/pr-9048 branch from e99e0d5 to d0c48ee Compare August 7, 2025 21:44

pkoutsovasilis removed the conflicts There is a conflict in the backported pull request label Aug 7, 2025

pkoutsovasilis added 2 commits August 8, 2025 16:04

fix: address beats breaking API func calls

9a2b39f

fix: mage fmt

21576ed

pkoutsovasilis approved these changes Aug 8, 2025

View reviewed changes

pkoutsovasilis merged commit b00cec3 into 8.19 Aug 8, 2025
19 checks passed

pkoutsovasilis deleted the mergify/bp/8.19/pr-9048 branch August 8, 2025 15:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[8.19] (backport #9048) ci: build agent from snapshot DRA#9280

[8.19] (backport #9048) ci: build agent from snapshot DRA#9280
pkoutsovasilis merged 6 commits into8.19from
mergify/bp/8.19/pr-9048

mergify bot commented Aug 7, 2025

Uh oh!

mergify bot commented Aug 7, 2025

Uh oh!

elasticmachine commented Aug 7, 2025

Uh oh!

pkoutsovasilis commented Aug 7, 2025

Uh oh!

pkoutsovasilis commented Aug 7, 2025

Uh oh!

elasticmachine commented Aug 7, 2025 •

edited

Loading

Uh oh!

elastic-sonarqube bot commented Aug 8, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

mergify bot commented Aug 7, 2025

What does this PR do?

Why is it important?

Checklist

Disruptive User Impact

How to test this PR locally

Related issues

Uh oh!

mergify bot commented Aug 7, 2025

Uh oh!

elasticmachine commented Aug 7, 2025

Uh oh!

pkoutsovasilis commented Aug 7, 2025

Uh oh!

pkoutsovasilis commented Aug 7, 2025

Uh oh!

elasticmachine commented Aug 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Build Failed

Failed CI Steps

History

Uh oh!

elastic-sonarqube bot commented Aug 8, 2025

Quality Gate passed

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

elasticmachine commented Aug 7, 2025 •

edited

Loading